Automated calibration of endoscopes with pull wires

ABSTRACT

A surgical robotic system automatically calibrates tubular and flexible surgical tools such as endoscopes. By compensating for unideal behavior of an endoscope, the surgical robotic system can accurately model motions of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. During calibration, the surgical robotic system moves the endoscope to a target position and receives data describing an actual position and/or orientation of the endoscope. The surgical robotic system determines gain values based at least on the discrepancy between the target position and the actual position. The endoscope can include tubular components referred to as a sheath and leader. An instrument device manipulator of the surgical robotic system actuates pull wires coupled to the sheath and/or the leader, which causes the endoscope to articulate.

CROSS-REFERENCE TO RELATED APPLICATIONS

The subject matter of the present application is related to U.S.application Ser. No. 14/523,760, filed on Oct. 24, 2014, entitled“SYSTEM FOR ROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS”,the entire disclosure of which is incorporated herein by reference.

BACKGROUND 1. Field of Art

This description generally relates to surgical robotics, andparticularly to an automated process for calibrating endoscopes withpull wires.

2. Description of the Related Art

Robotic technologies have a range of applications. In particular,robotic arms help complete tasks that a human would normally perform.For example, factories use robotic arms to manufacture automobiles andconsumer electronics products. Additionally, scientific facilities userobotic arms to automate laboratory procedures such as transportingmicroplates. Recently, physicians have started using robotic arms tohelp perform surgical procedures. For instance, physicians use roboticarms to control surgical instruments such as endoscopes.

Endoscopes with movable tips help perform surgical procedures in aminimally invasive manner. A movable tip can be directed to a remotelocation of a patient, such as the lung or blood vessel. Deviation ofthe tip's actual position from a target position may result inadditional manipulation to correct the tip's position. Existingtechniques for manual calibration may rely on limited amounts ofendoscope tip deflection that does not accurately model motions of thetip.

SUMMARY

A surgical robotic system automatically calibrates tubular and flexiblesurgical tools such as endoscopes. By compensating for unideal behaviorof an endoscope, the surgical robotic system can accurately modelmotions of the endoscope and navigate the endoscope while performing asurgical procedure on a patient. During calibration, the surgicalrobotic system moves the endoscope to a target position and receivescalibration data describing an actual position of the endoscope. Thesurgical robotic system determines the actual position and/ororientation that the endoscope moves in response to commands based oncalibration data captured by spatial sensors. Example spatial sensorsinclude accelerometers, gyroscopes, electromagnetic sensors, opticalfibers, cameras, and fluoroscopic imaging systems. The surgical roboticsystem determines gain values based at least on the discrepancy betweenthe target position and the actual position. The surgical robotic systemcan perform calibration before or during a surgical procedure.

In some embodiments, an endoscope includes tubular components referredto as a sheath and leader. The gain values may also be based on a lengthof the leader extending out of the sheath, or a relative roll angle ofthe leader relative to the sheath. The surgical robotic system moves thesheath and leader using an instrument device manipulator (IDM). Forexample, the IDM translates pull wires coupled to the sheath or theleader, which causes the endoscope to move along different axis, e.g., apitch, yaw, and roll axis.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a surgical robotic system according to oneembodiment.

FIG. 2 illustrates a command console for a surgical robotic systemaccording to one embodiment.

FIG. 3A illustrates multiple degrees of motion of an endoscope accordingto one embodiment.

FIG. 3B is a top view of an endoscope including sheath and leadercomponents according to one embodiment.

FIG. 3C is a cross sectional side view of a sheath of an endoscopeaccording to one embodiment.

FIG. 3D is an isometric view of a helix section of a sheath of anendoscope according to one embodiment.

FIG. 3E is another isometric view of a helix section of a sheath of anendoscope according to one embodiment.

FIG. 3F is a side view of a sheath of an endoscope with a helix sectionaccording to one embodiment.

FIG. 3G is another view of the sheath of the endoscope shown in FIG. 3Faccording to one embodiment.

FIG. 3H is a cross sectional side view of a leader of an endoscopeaccording to one embodiment.

FIG. 3I is a cross sectional isometric view of a distal tip of theleader of the endoscope shown in FIG. 3H according to one embodiment.

FIG. 4A is an isometric view of an instrument device manipulator of asurgical robotic system according to one embodiment.

FIG. 4B is an exploded isometric view of the instrument devicemanipulator shown in FIG. 4A according to one embodiment.

FIG. 4C is an isometric view of an independent drive mechanism of theinstrument device manipulator shown in FIG. 4A according to oneembodiment.

FIG. 4D illustrates a conceptual diagram that shows how forces may bemeasured by a strain gauge of the independent drive mechanism shown inFIG. 4C according to one embodiment.

FIG. 5A illustrates pull wires inside an endoscope according to oneembodiment.

FIG. 5B shows a back view of an endoscope in a resting positionaccording to one embodiment.

FIG. 5C shows a top view of the endoscope shown in FIG. 5B according toone embodiment.

FIG. 5D shows a side view of the endoscope shown in FIG. 5B according toone embodiment.

FIG. 5E shows a back view of the endoscope shown in FIG. 5B in adeflected position according to one embodiment.

FIG. 5F shows a top view of the endoscope shown in FIG. 5E according toone embodiment.

FIG. 5G shows a side view of the endoscope shown in FIG. 5E according toone embodiment.

FIG. 5H shows a back view of the endoscope shown in FIG. 5B in adeflected position with an additional unideal offset according to oneembodiment.

FIG. 5I shows a top view of the endoscope shown in FIG. 5H according toone embodiment.

FIG. 5J shows a back view of the endoscope shown in FIG. 5B in a restingposition according to one embodiment.

FIG. 5K shows a side view of the endoscope shown in FIG. 5J according toone embodiment.

FIG. 5L shows a back view of the endoscope shown in FIG. 5J in adeflected position with an additional unideal roll offset according toone embodiment.

FIG. 5M shows a side view of the endoscope shown in FIG. 5L according toone embodiment.

FIG. 6A is a diagram of an electromagnetic tracking system according toone embodiment.

FIG. 6B is a diagram of cameras in proximity to an endoscope accordingto one embodiment.

FIG. 6C is a diagram of motion tracking cameras in proximity to anendoscope including fiducial markers according to one embodiment.

FIG. 6D is a diagram of an endoscope with a shape sensing optical fiberaccording to one embodiment.

FIG. 6E is a diagram of a fluoroscopic imaging system in proximity to anendoscope according to one embodiment.

FIG. 7A shows a length of a leader of an endoscope extended outside of asheath of the endoscope according to one embodiment.

FIG. 7B shows a relative roll angle of the leader of the endoscoperelative to the sheath of the endoscope according to one embodiment.

FIG. 8A is a flowchart of a process for automated calibration of anendoscope according to one embodiment.

FIG. 8B is a flowchart of a process for automated calibration of anendoscope based on length of extension and relative roll angle accordingto one embodiment.

FIG. 9 is a flowchart of a process for intraoperative automatedcalibration of an endoscope to one embodiment.

The figures depict embodiments of the present invention for purposes ofillustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION

The methods and apparatus disclosed herein are well suited for use withone or more endoscope components or steps as described in U.S.application Ser. No. 14/523,760, filed on Oct. 24, 2014, published asU.S. Pat. Pub. No. US 2015/0119637, entitled “SYSTEM FORROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS,” the fulldisclosure of which has been previously incorporated by reference. Theaforementioned application describes system components, endolumenalsystems, virtual rail configurations, mechanism changer interfaces,instrument device manipulators (IDMs), endoscope tool designs, controlconsoles, endoscopes, instrument device manipulators, endolumenalnavigation, and endolumenal procedures suitable for combination inaccordance with embodiments disclosed herein.

I. Surgical Robotic System

FIG. 1 illustrates a surgical robotic system 100 according to oneembodiment. The surgical robotic system 100 includes a base 101 coupledto one or more robotic arms, e.g., robotic arm 102. The base 101 iscommunicatively coupled to a command console, which is further describedwith reference to FIG. 2 in Section II. Command Console. The base 101can be positioned such that the robotic arm 102 has access to perform asurgical procedure on a patient, while a user such as a physician maycontrol the surgical robotic system 100 from the comfort of the commandconsole. In some embodiments, the base 101 may be coupled to a surgicaloperating table or bed for supporting the patient. Though not shown inFIG. 1 for purposes of clarity, the base 101 may include subsystems suchas control electronics, pneumatics, power sources, optical sources, andthe like. The robotic arm 102 includes multiple arm segments 110 coupledat joints 111, which provides the robotic arm 102 multiple degrees offreedom, e.g., seven degrees of freedom corresponding to seven armsegments. The base 101 may contain a source of power 112, pneumaticpressure 113, and control and sensor electronics 114—includingcomponents such as a central processing unit, data bus, controlcircuitry, and memory—and related actuators such as motors to move therobotic arm 102. The electronics 114 in the base 101 may also processand transmit control signals communicated from the command console.

In some embodiments, the base 101 includes wheels 115 to transport thesurgical robotic system 100. Mobility of the surgical robotic system 100helps accommodate space constraints in a surgical operating room as wellas facilitate appropriate positioning and movement of surgicalequipment. Further, the mobility allows the robotic arms 102 to beconfigured such that the robotic arms 102 do not interfere with thepatient, physician, anesthesiologist, or any other equipment. Duringprocedures, a user may control the robotic arms 102 using controldevices such as the command console.

In some embodiments, the robotic arm 102 includes set up joints that usea combination of brakes and counter-balances to maintain a position ofthe robotic arm 102. The counter-balances may include gas springs orcoil springs. The brakes, e.g., fail safe brakes, may be includemechanical and/or electrical components. Further, the robotic arms 102may be gravity-assisted passive support type robotic arms.

Each robotic arm 102 may be coupled to an instrument device manipulator(IDM) 117 using a mechanism changer interface (MCI) 116. The IDM 117 canbe removed and replaced with a different type of IDM, for example, afirst type of IDM manipulates an endoscope, while a second type of IDMmanipulates a laparoscope. The MCI 116 includes connectors to transferpneumatic pressure, electrical power, electrical signals, and opticalsignals from the robotic arm 102 to the IDM 117. The MCI 116 can be aset screw or base plate connector. The IDM 117 manipulates surgicalinstruments such as the endoscope 118 using techniques including directdrive, harmonic drive, geared drives, belts and pulleys, magneticdrives, and the like. The MCI 116 is interchangeable based on the typeof IDM 117 and can be customized for a certain type of surgicalprocedure. The robotic 102 arm can include a joint level torque sensingand a wrist at a distal end, such as the KUKA AG® LBR5 robotic arm.

The endoscope 118 is a tubular and flexible surgical instrument that isinserted into the anatomy of a patient to capture images of the anatomy(e.g., body tissue). In particular, the endoscope 118 includes one ormore imaging devices (e.g., cameras or sensors) that capture the images.The imaging devices may include one or more optical components such asan optical fiber, fiber array, or lens. The optical components movealong with the tip of the endoscope 118 such that movement of the tip ofthe endoscope 118 results in changes to the images captured by theimaging devices. The endoscope 118 is further described with referenceto FIGS. 3A-I in Section III. Endoscope.

Robotic arms 102 of the surgical robotic system 100 manipulate theendoscope 118 using elongate movement members. The elongate movementmembers may include pull wires, also referred to as pull or push wires,cables, fibers, or flexible shafts. For example, the robotic arms 102actuate multiple pull wires coupled to the endoscope 118 to deflect thetip of the endoscope 118. The pull wires may include both metallic andnon-metallic materials such as stainless steel, Kevlar, tungsten, carbonfiber, and the like. The endoscope 118 may exhibit unideal behavior inresponse to forces applied by the elongate movement members. The unidealbehavior may be due to imperfections or variations in stiffness andcompressibility of the endoscope 118, as well as variability in slack orstiffness between different elongate movement members.

The surgical robotic system 100 includes a computer system 120, forexample, a computer processor. The computer system 120 includes acalibration module 130, calibration store 140, command module 150, anddata processing module 160. The data processing module 160 can processcalibration data collected by the surgical robotic system 100. Thecalibration module 130 can characterize the unideal behavior of theendoscope 118 using gain values based on the calibration data. Thecomputer system 120 and its modules are further described in SectionVII: Calibration Process Flows. The surgical robotic system 100 can moreaccurately control an endoscope 118 by determining accurate values ofthe gain values. In some embodiments, some or all functionality of thecomputer system 120 is performed outside the surgical robotic system100, for example, on another computer system or server communicativelycoupled to the surgical robotic system 100.

II. Command Console

FIG. 2 illustrates a command console 200 for a surgical robotic system100 according to one embodiment. The command console 200 includes aconsole base 201, display modules 202, e.g., monitors, and controlmodules, e.g., a keyboard 203 and joystick 204. In some embodiments, oneor more of the command module 200 functionality may be integrated into abase 101 of the surgical robotic system 100 or another systemcommunicatively coupled to the surgical robotic system 100. A user 205,e.g., a physician, remotely controls the surgical robotic system 100from an ergonomic position using the command console 200.

The console base 201 may include a central processing unit, a memoryunit, a data bus, and associated data communication ports that areresponsible for interpreting and processing signals such as cameraimagery and tracking sensor data, e.g., from the endoscope 118 shown inFIG. 1. In some embodiments, both the console base 201 and the base 101perform signal processing for load-balancing. The console base 201 mayalso process commands and instructions provided by the user 205 throughthe control modules 203 and 204. In addition to the keyboard 203 andjoystick 204 shown in FIG. 2, the control modules may include otherdevices, for example, computer mice, trackpads, trackballs, controlpads, video game controllers, and sensors (e.g., motion sensors orcameras) that capture hand gestures and finger gestures.

The user 205 can control a surgical instrument such as the endoscope 118using the command console 200 in a velocity mode or position controlmode. In velocity mode, the user 205 directly controls pitch and yawmotion of a distal end of the endoscope 118 based on direct manualcontrol using the control modules. For example, movement on the joystick204 may be mapped to yaw and pitch movement in the distal end of theendoscope 118. The joystick 204 can provide haptic feedback to the user205. For example, the joystick 204 vibrates to indicate that theendoscope 118 cannot further translate or rotate in a certain direction.The command console 200 can also provide visual feedback (e.g., pop-upmessages) and/or audio feedback (e.g., beeping) to indicate that theendoscope 118 has reached maximum translation or rotation.

In position control mode, the command console 200 uses athree-dimensional (3D) map of a patient and pre-determined computermodels of the patient to control a surgical instrument, e.g., theendoscope 118. The command console 200 provides control signals torobotic arms 102 of the surgical robotic system 100 to manipulate theendoscope 118 to a target location. Due to the reliance on the 3D map,position control mode requires accurate mapping of the anatomy of thepatient.

In some embodiments, users 205 can manually manipulate robotic arms 102of the surgical robotic system 100 without using the command console200. During setup in a surgical operating room, the users 205 may movethe robotic arms 102, endoscopes 118, and other surgical equipment toaccess a patient. The surgical robotic system 100 may rely on forcefeedback and inertia control from the users 205 to determine appropriateconfiguration of the robotic arms 102 and equipment.

The display modules 202 may include electronic monitors, virtual realityviewing devices, e.g., goggles or glasses, and/or other means of displaydevices. In some embodiments, the display modules 202 are integratedwith the control modules, for example, as a tablet device with atouchscreen. Further, the user 205 can both view data and input commandsto the surgical robotic system 100 using the integrated display modules202 and control modules.

The display modules 202 can display 3D images using a stereoscopicdevice, e.g., a visor or goggle. The 3D images provide an “endo view”(i.e., endoscopic view), which is a computer 3D model illustrating theanatomy of a patient. The “endo view” provides a virtual environment ofthe patient's interior and an expected location of an endoscope 118inside the patient. A user 205 compares the “endo view” model to actualimages captured by a camera to help mentally orient and confirm that theendoscope 118 is in the correct—or approximately correct—location withinthe patient. The “endo view” provides information about anatomicalstructures, e.g., the shape of an intestine or colon of the patient,around the distal end of the endoscope 118. The display modules 202 cansimultaneously display the 3D model and computerized tomography (CT)scans of the anatomy the around distal end of the endoscope 118.Further, the display modules 202 may overlay pre-determined optimalnavigation paths of the endoscope 118 on the 3D model and CT scans.

In some embodiments, a model of the endoscope 118 is displayed with the3D models to help indicate a status of a surgical procedure. Forexample, the CT scans identify a lesion in the anatomy where a biopsymay be necessary. During operation, the display modules 202 may show areference image captured by the endoscope 118 corresponding to thecurrent location of the endoscope 118. The display modules 202 mayautomatically display different views of the model of the endoscope 118depending on user settings and a particular surgical procedure. Forexample, the display modules 202 show an overhead fluoroscopic view ofthe endoscope 118 during a navigation step as the endoscope 118approaches an operative region of a patient.

III. Endoscope

FIG. 3A illustrates multiple degrees of motion of an endoscope 118according to one embodiment. The endoscope 118 is an embodiment of theendoscope 118 shown in FIG. 1. As shown in FIG. 3A, the tip 301 of theendoscope 118 is oriented with zero deflection relative to alongitudinal axis 306 (also referred to as a roll axis 306). To captureimages at different orientations of the tip 301, a surgical roboticsystem 100 deflects the tip 301 on a positive yaw axis 302, negative yawaxis 303, positive pitch axis 304, negative pitch axis 305, or roll axis306. The tip 301 or body 310 of the endoscope 118 may be elongated ortranslated in the longitudinal axis 306, x-axis 308, or y-axis 309.

FIG. 3B is a top view of an endoscope 118 including sheath and leadercomponents according to one embodiment. The endoscope 118 includes aleader 315 tubular component nested or partially nested inside andlongitudinally-aligned with a sheath 311 tubular component. The sheath311 includes a proximal sheath section 312 and distal sheath section313. The leader 315 has a smaller outer diameter than the sheath 311 andincludes a proximal leader section 316 and distal leader section 317.The sheath base 314 and the leader base 318 actuate the distal sheathsection 313 and the distal leader section 317, respectively, forexample, based on control signals from a user of a surgical roboticsystem 100. The sheath base 314 and the leader base 318 are, e.g., partof the IDM 117 shown in FIG. 1. The construction, composition,capabilities, and use of distal leader section 317, which may also bereferred to as a flexure section, are disclosed in U.S. patentapplication Ser. No. 14/201,610, filed Mar. 7, 2014, and U.S. patentapplication Ser. No. 14/479,095, filed Sep. 5, 2014, the entire contentsof which are incorporated by reference.

Both the sheath base 314 and the leader base 318 include drivemechanisms (e.g., the independent drive mechanism further described withreference to FIG. 4A-D in Section III. D. Instrument Device Manipulator)to control pull wires coupled to the sheath 311 and leader 315. Forexample, the sheath base 314 generates tensile loads on pull wirescoupled to the sheath 311 to deflect the distal sheath section 313.Similarly, the leader base 318 generates tensile loads on pull wirescoupled to the leader 315 to deflect the distal leader section 317. Boththe sheath base 314 and leader base 318 may also include couplings forthe routing of pneumatic pressure, electrical power, electrical signals,or optical signals from IDMs to the sheath 311 and leader 314,respectively. A pull wire may include a steel coil pipe along the lengthof the pull wire within the sheath 311 or the leader 315, whichtransfers axial compression back to the origin of the load, e.g., thesheath base 314 or the leader base 318, respectively.

The endoscope 118 can navigate the anatomy of a patient with ease due tothe multiple degrees of freedom provided by pull wires coupled to thesheath 311 and the leader 315. For example, four or more pull wires maybe used in either the sheath 311 and/or the leader 315, providing eightor more degrees of freedom. In other embodiments, up to three pull wiresmay be used, providing up to six degrees of freedom. The sheath 311 andleader 315 may be rotated up to 360 degrees along a longitudinal axis306, providing more degrees of motion. The combination of rotationalangles and multiple degrees of freedom provides a user of the surgicalrobotic system 100 with a user friendly and instinctive control of theendoscope 118.

III. A. Endoscope Sheath

FIG. 3C is a cross sectional side view of the sheath 311 of theendoscope 118 according to one embodiment. The sheath 311 includes alumen 323 sized to accommodate a tubular component such as the leader315 shown in FIG. 3B. The sheath 311 includes walls 324 with pull wires325 and 326 running through conduits 327 and 328 inside the length ofwalls 324. The conduits include a helix section 330 and a distalnon-helix section 329. Appropriate tensioning of pull wire 325 maycompress the distal end 320 in the positive y-axis direction, whileminimizing bending of the helix section 330. Similarly, appropriatetensioning of pull wire 326 may compress distal end 320 in the negativey-axis direction. In some embodiments, the lumen 323 is not concentricwith the sheath 311.

Pull wires 325 and 326 do not necessarily run straight through thelength of sheath 311. Rather, the pull wires 325 and 326 spiral aroundsheath 311 along helix section 330 and run longitudinally straight(i.e., approximately parallel to the longitudinal axis 306) along thedistal non-helix section 329 and any other non-helix section of thesheath 311. The helix section 330 may start and end anywhere along thelength of the sheath 311. Further, the length and pitch of helix section330 may be determined based on desired properties of sheath 311, e.g.,flexibility of the sheath 311 and friction in the helix section 330.

Though the pull wires 325 and 326 are positioned at 180 degrees relativeto each other in FIG. 3C, it should be noted that pull wires of thesheath 311 may be positioned at different angles. For example, threepull wires of a sheath may each be positioned at 120 degrees relative toeach other. In some embodiments, the pull wires are not equally spacedrelative to each other, i.e., without a constant angle offset.

III. B. Helix Sections

FIG. 3D is an isometric view of a helix section 330 of the sheath 311 ofthe endoscope 118 according to one embodiment. FIG. 3D shows only onepull wire 325 for the purpose of distinguishing between the distalnon-helix section 329 and the helix section 330. In some embodiments,the helix section 330 has a variable pitch.

FIG. 3E is another isometric view of a helix section 330 of a sheath 311of an endoscope 118 according to one embodiment. FIG. 3E shows four pullwires 325, 326, 351, and 352 extending along the distal non-helixsection 329 and the variable pitch helix section 330.

Helix sections 330 in the sheath 311 and leader 315 of the endoscope 118help a surgical robotic system 100 and/or a user navigate the endoscope118 through non-linear pathways in the anatomy of a patient, e.g.,intestines or the colon. When navigating the nonlinear pathways, it isuseful for the endoscope 118 to remain flexible, while still having acontrollable distal section (in both the sheath 311 and the leader 315).Further, it is advantageous to reduce the amount of unwanted bendingalong the endoscope 118. In previous endoscope designs, tensioning thepull wires to manipulate the distal section generated the unwantedbending and torquing along a length of the endoscope, which may bereferred to as muscling and curve alignment, respectively.

FIG. 3F is a side view of the sheath 311 of the endoscope 118 with ahelix section 330 according to one embodiment. FIGS. 3F-G illustrate howthe helix section 330 helps substantially mitigate muscling and curvealignment. Since the pull wire 325 is spiraled around the length ofhelix section 330, the pull wire 325 radially and symmetricallydistributes a compressive load 335 in multiple directions around thelongitudinal axis 306. Further, bending moments imposed on the endoscope118 are also symmetrically distributed around the longitudinal axis 306,which counterbalances and offsets opposing compressive forces andtensile forces. The distribution of the bending moments results inminimal net bending and rotational forces, creating a low potentialenergy state of the endoscope 118, and thus eliminating or substantiallymitigating muscling and curve alignment.

The pitch of the helix section 330 can affect the friction and thestiffness of the helix section 330. For example, the helix section 330may be shorter to allow for a longer distal non-helix section 329,resulting in less friction and/or stiffness of the helix section 330.

FIG. 3G is another view of the sheath 311 of the endoscope 118 shown inFIG. 3F according to one embodiment. Compared to the distal non-helixsection 329 shown in FIG. 3F, the distal non-helix section 329 shown inFIG. 3G is deflected at a greater angle.

III. C. Endoscope Leader

FIG. 3H is a cross sectional side view of the leader 315 of theendoscope 118 according to one embodiment. The leader 315 includes atleast one working channel 343 and pull wires 344 and 345 running throughconduits 341 and 342, respectively, along the length of the walls 348.The pull wires 344 and 345 and conduits 341 and 342 are substantiallythe same as the pull wires 325 and 326 and the conduits 327 and 328 inFIG. 3C, respectively. For example, the pull wires 344 and 345 may havea helix section that helps mitigate muscling and curve alignment of theleader 315, similar to the sheath 311 as previously described.

FIG. 3I is a cross sectional isometric view of a distal tip of theleader 315 of the endoscope 118 shown in FIG. 3H according to oneembodiment. The leader 315 includes an imaging device 349 (e.g.,charge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) camera, imaging fiber bundle, etc.), light sources 350 (e.g.,light-emitting diode (LED), optic fiber, etc.), at least two pull wires344 and 345, and at least one working channel 343 for other components.For example, other components include camera wires, an insufflationdevice, a suction device, electrical wires, fiber optics, an ultrasoundtransducer, electromagnetic (EM) sensing components, and opticalcoherence tomography (OCT) sensing components. In some embodiments, theleader 315 includes a pocket hole to accommodate insertion of acomponent into a working channel 343. As shown in FIG. 3I, the pullwires 344 and 345 are not concentric with the an imaging device 349 orthe working channel 343.

III. D. Instrument Device Manipulator

FIG. 4A is an isometric view of an instrument device manipulator 117 ofthe surgical robotic system 100 according to one embodiment. The roboticarm 102 is coupled to the IDM 117 via an articulating interface 401. TheIDM 117 is coupled to the endoscope 118. The articulating interface 401may transfer pneumatic pressure, power signals, control signals, andfeedback signals to and from the robotic arm 102 and the IDM 117. TheIDM 117 may include a gear head, motor, rotary encoder, power circuits,and control circuits. A tool base 403 for receiving control signals fromthe IDM 117 is coupled to the proximal end of the endoscope 118. Basedon the control signals, the IDM 117 manipulates the endoscope 118 byactuating output shafts, which are further described below withreference to FIG. 4B.

FIG. 4B is an exploded isometric view of the instrument devicemanipulator shown in FIG. 4A according to one embodiment. In FIG. 4B,the endoscopic 118 has been removed from the IDM 117 to reveal theoutput shafts 405, 406, 407, and 408.

FIG. 4C is an isometric view of an independent drive mechanism of theinstrument device manipulator 117 shown in FIG. 4A according to oneembodiment. The independent drive mechanism can tighten or loosen thepull wires 421, 422, 423, and 424 (e.g., independently from each other)of an endoscope by rotating the output shafts 405, 406, 407, and 408 ofthe IDM 117, respectively. Just as the output shafts 405, 406, 407, and408 transfer force down pull wires 421, 422, 423, and 424, respectively,through angular motion, the pull wires 421, 422, 423, and 424 transferforce back to the output shafts. The IDM 117 and/or the surgical roboticsystem 100 can measure the transferred force using a sensor, e.g., astrain gauge further described below.

FIG. 4D illustrates a conceptual diagram that shows how forces may bemeasured by a strain gauge 434 of the independent drive mechanism shownin FIG. 4C according to one embodiment. A force 431 may directed awayfrom the output shaft 405 coupled to the motor mount 433 of the motor437. Accordingly, the force 431 results in horizontal displacement ofthe motor mount 433. Further, the strain gauge 434 horizontally coupledto the motor mount 433 experiences strain in the direction of the force431. The strain may be measured as a ratio of the horizontaldisplacement of the tip 435 of strain gauge 434 to the overallhorizontal width 436 of the strain gauge 434.

In some embodiments, the IDM 117 includes additional sensors, e.g.,inclinometers or accelerometers, to determine an orientation of the IDM117. Based on measurements from the additional sensors and/or the straingauge 434, the surgical robotic system 100 can calibrate readings fromthe strain gauge 434 to account for gravitational load effects. Forexample, if the IDM 117 is oriented on a horizontal side of the IDM 117,the weight of certain components of the IDM 117 may cause a strain onthe motor mount 433. Accordingly, without accounting for gravitationalload effects, the strain gauge 434 may measure strain that did notresult from strain on the output shafts.

IV. Unideal Endoscope Motion

FIG. 5A illustrates pull wires inside the endoscope 118 according to oneembodiment. The endoscope 118 may include a different number of pullwires depending upon the construction of the endoscope, but for sake ofexample the following description assumes a construction where theendoscope 118 includes the four pull wires 421, 422, 423, and 423 each acorresponding to a direction of movement along the yaw 510 and pitch 520axis. In particular, pulling the pull wires 421, 422, 423, and 423 movesthe endoscope 118 in the positive pitch direction, positive yawdirection, negative pitch direction, and negative yaw direction,respectively. Though the pull wires shown in FIG. 5A are each aligned toa yaw 510 or pitch 520 direction, in other embodiments, the pull wiresmay not necessarily be aligned along these axes, the axes above arearbitrarily chosen for convenience of explanation. For example, a pullwire may be aligned with (e.g., intersect) the point 540 in theendoscope 118. Thus, translating the pull wire would cause the endoscope118 to move in both the yaw 510 and pitch 520 directions. In the exampleembodiment described throughout, when the endoscope 118 is in a restingposition the pull wires are approximately parallel with the roll 530axis.

The endoscope 118 may include one or more spatial sensors 550 coupledtoward the distal tip of the endoscope 118. The spatial sensors 550 canbe, for example, an electromagnetic (EM) sensor, accelerometer,gyroscope, fiducial marker, and/or other types of sensors. In oneembodiment, the spatial sensor 550 is a shape sensing optical fiberembedded inside the endoscope 118 and running along a length of theendoscope 118. The spatial sensors 550 may provide spatial dataindicating a position and/or orientation of the endoscope 118, e.g., inreal-time. Spatial data may also be used as calibration data to assistin calibration of the endoscope 118.

In an ideal endoscope, translating pull wires of the endoscope moves theendoscope exactly to a target position or orientation, e.g., bend thetip of the endoscope 90 degrees in the positive pitch direction.However, in practice, due to imperfections of the endoscope, the targetmotion does not necessarily match the actual motion of the endoscope,and the endoscope may exhibit nonlinear behavior. Imperfections mayarise for a variety of reasons, examples of which may be the result ofdefects in manufacturing (e.g., a pull wire is not properly aligned withan axis of motion), variances in the pull wires (e.g., a pull wire ismore stiff, or different in length, than another pull wire), orvariances in the endoscope material (e.g., the pitch direction bendsmore easily than the yaw direction).

IV. A. Unideal Offset in Pitch Direction

FIGS. 5B-5D illustrate three views of an endoscope 118 in a restingposition. FIGS. 5E-G illustrate the same three views after the endoscope118 has been moved to a deflected position in response to a command toarticulate to a target deflection of 90 degrees in the positive pitch520 direction. As shown in FIGS. 5E-G, the actual deflection of theendoscope 118 exhibits an unideal offset in the positive pitch 520direction.

FIG. 5B shows a back view of the endoscope 118 in a resting positionaccording to one embodiment. In the back view, a viewer is looking downfrom the proximal end of the endoscope, where the opposite distal endwould be inserted into a body of a patient. The cross section of theendoscope 118 is aligned to the origin of the yaw 510 and pitch 520axis. The endoscope 118 is parallel to the roll 530 axis, and a spatialsensor 550 is coupled toward the tip of the endoscope 118. FIG. 5C showsa top view of the endoscope 118 shown in FIG. 5B according to oneembodiment. As an illustrative example, a patient is lying horizontallyflat on a table for a surgical procedure. The endoscope 118 ispositioned to be parallel to the body of the patient, and the surgicalrobotic system 100 inserts the endoscope 118 into the body whilemaintaining the parallel configuration. In the top view, a viewer islooking down from above the body of the patient. FIG. 5D shows a sideview of the endoscope 118 shown in FIG. 5B according to one embodiment.

FIG. 5E shows a back view of the endoscope 118 shown in FIG. 5B in thedeflected position according to one embodiment. FIG. 5F shows a top viewof the endoscope 118 shown in FIG. 5E according to one embodiment. FIG.5G shows a side view of the endoscope 118 shown in FIG. 5E according toone embodiment. The dotted outline of the endoscope 118 indicates thetarget deflected position that the endoscope should have moved to inresponse to the command, e.g., the tip of the endoscope 118 is supposeddeflect 90 degrees in the positive pitch direction to become parallel tothe pitch 520 axis. However, the actual deflected position is short of a90 degree deflection, and thus exhibits the unideal offset in thepositive pitch 520 direction.

IV. B. Unideal Offset in Yaw Direction

FIG. 5H shows a back view of the endoscope 118 shown in FIG. 5B in adeflected position with an additional unideal offset according to oneembodiment. In particular, in addition to the unideal offset in thepositive pitch 520 direction, the endoscope shown in FIG. 5H exhibits anadditional unideal offset in the positive yaw 510 direction. Thus, thedistal end (e.g., tip) of the endoscope 118 is “curved,” in contrast tothe distal end of the endoscope shown in FIG. 5F that is “straight.” Theendoscope 118 shown in FIG. 5H has imperfections in two directions(positive pitch and yaw), however, in other embodiments, endoscopes canexhibit imperfections in any number of directions (e.g., negative pitchand yaw, as well as roll).

FIG. 5I shows a top view of the endoscope 118 shown in FIG. 5H accordingto one embodiment.

IV. C. Unideal Offset in Roll Direction

FIGS. 5J-5K illustrate two views of an endoscope 118 in a restingposition. Four markers 560 are shown on the endoscope 118 for purposesof illustrating the alignment of the endoscope 118 relative to the yaw510 and pitch 520 directions. In the resting position, each of themarkers are aligned with the yaw 510 and pitch 520 axis.

FIGS. 5L-M illustrate the same two views after the endoscope 118 hasbeen moved to a deflected position in response to a command toarticulate to a target deflection of 90 degrees in the positive pitch520 direction. As shown in FIGS. 5L-M, the actual deflection of theendoscope 118 exhibits an unideal offset in the roll 530 direction (andno other unideal offsets in this example). The dotted outline of theendoscope 118 indicates the target deflected position that the endoscopeshould have moved to in response to the command, e.g., the tip of theendoscope 118 is supposed deflect 90 degrees to become parallel to thepitch 520 axis. The actual deflected position has a deflection 90degrees, but also has a rotation along the roll 530 axis. Thus, the fourmarkers 560 are no longer aligned with the yaw 510 and pitch 520 axis.Similar to the endoscope shown in FIG. 5E, the distal end of theendoscope in FIG. 5L is “straight” and not “curved.” In someembodiments, rotation of the proximal end of an endoscope is accompaniedby corresponding rotation of the distal end of the endoscope (and viceversa). The rotations may be equal or different, e.g., a 10 degree rolloffset of the proximal end causes a 20 degree roll offset of the distalend. As another example, there may be no roll offset at the proximalend, and a nonzero roll offset at the distal end.

FIG. 5M shows a side view of the endoscope shown in FIG. 5L according toone embodiment.

In summary, FIGS. 5B-G illustrate an unideal offset in the positivepitch direction, FIGS. 5H-I illustrate an unideal offset in the yawdirection in addition to in the positive pitch direction, and FIGS. 5J-Millustrate an unideal offset in the roll direction. In otherembodiments, endoscopes may exhibit unideal offsets in any number orcombination of directions. The magnitude of the offsets may vary betweendifferent directions.

V. Spatial Sensors

V. A. Electromagnetic Sensors

FIG. 6A is a diagram of electromagnetic tracking system according to oneembodiment. The spatial sensor 550 coupled to the tip of the endoscope118 includes one or more EM sensors 550 that detect an electromagneticfield (EMF) generated by one or more EMF generators 600 in proximity tothe endoscope 118. The strength of the detected EMF is a function of theposition and/or orientation of the endoscope 118. If the endoscope 118includes more than one EM sensor 550, for example, a first EM sensor iscoupled to a leader tubular component and a second EM sensor is coupledto a sheath tubular component of the endoscope 118.

One or more EMF generators 600 are located externally to a patient. TheEMF generators 600 emit EM fields that are picked up by the EM sensor550.

If multiple EMF generators 600 and/or EM sensors 550 are used, they maybe modulated in a number of different ways so that when theiremitted/received fields are processed by the computer system 120 (or anycomputer system external to the surgical robotic system 100), thesignals are separable. Thus, the computer system 120 can processmultiple signals (sent and/or received) each as a separate inputproviding separate triangulation location regarding the location of theEM sensor/s 550, and by extension the position of the endoscope 118. Forexample, multiple EMF generators 600 may be modulated in time or infrequency, and may use orthogonal modulations so that each signal isfully separable from each other signal (e.g., using signal processingtechniques such as filtering and Fourier Transforms) despite possiblyoverlapping in time. Further, the multiple EM sensors 550 and/or EMFgenerators 600 may be oriented relative to each other in Cartesian spaceat non-zero, non-orthogonal angles so that changes in orientation of theEM sensor/s 550 will result in at least one of the EM sensor/s 550receiving at least some signal from the one or more EMF generators 600at any instant in time. For example, each EMF generator 600 may be,along any axis, offset at a small angle (e.g., 7 degrees) from each oftwo other EM generators 600 (and similarly with multiple EM sensors 550s). As many EMF generators or EM sensors as desired may be used in thisconfiguration to assure accurate EM sensor position information alongall three axes and, if desired, at multiple points along the endoscope118.

V. B. Camera Sensors

FIG. 6B is a diagram of cameras in proximity to an endoscope 118according to one embodiment. The cameras may include any type of opticalcameras such as digital video cameras, stereo cameras, high-speedcameras, light field cameras, etc. A first camera 610 is parallel to alongitudinal axis of the endoscope 118. A second camera 620 isorthogonal to the first camera 610. Since the cameras each capture imageframes showing the position and/or orientation of the endoscope in atleast two-dimensions, aligning the two cameras orthogonal to each otherenables the surgical robotic system 100 to receive information about theendoscope in at least three-dimensions (e.g., corresponding to thepitch, yaw, and roll axis). In other embodiments, three or more camerasmay be used to capture images of the endoscope 118. The data processingmodule 160 may implement object tracking image processing techniquesusing the captured image frames to determine the real-time 3D positionof the endoscope 118. Example techniques include correlation-basedmatching methods, feature-based methods, and optical flow.

FIG. 6C is a diagram of motion cameras in proximity to an endoscope 118including fiducial markers according to one embodiment. The spatialsensors 550 coupled to toward the distal end of the endoscope 118 arefiducial markers. The motion cameras 630 and 640 capture image framesthat track the position and movement of the fiducial markers. Though twofiducial markers and two motion cameras are shown in FIG. 6C, otherembodiments may include any other number of fiducial markers coupled tothe endoscope and/or motion cameras to track the fiducial markers.

In contrast to the cameras in FIG. 6B, the motion cameras in FIG. 6Ccapture data describing the motion of the fiducial markers. Thus, insome embodiments, the data processing module 160 requires lesscomputational resources to process the motion camera data than toprocess data captured from other optical cameras without using fiducialmarkers. For example, other optical cameras capture data describing thevisual appearance (e.g., color and size) of the endoscope 118. However,the motion cameras may only need to capture the real-time coordinateposition of each fiducial marker, which is sufficient for the dataprocessing module 160 to use to determine overall movement of theendoscope 118 in different directions (e.g., pitch, yaw, and roll) inresponse to commands from the surgical robotic system 100.

Camera based sensors may be more suitable for determining the positionof an endoscope outside a body of a patient, while EM sensors may bemore suitable for use cases where the endoscope is inside the body. Insome embodiments, image processing techniques using camera data providemore accurate or higher resolution position and motion data than EMsensor based techniques, e.g., because a user viewing the endoscopeoutside of the body can validate the results of image processingtechniques. In contrast, EM sensors have an advantage in that they canstill detect EM fields generated by EMF generators even when theendoscope is located inside a patient.

V. C. Shape Sensing Optical Fiber

FIG. 6D is a diagram of an endoscope 118 with a shape sensing opticalfiber according to one embodiment. The spatial sensor 550 is a shapesensing optical fiber embedded inside the endoscope 118. A console 650positioned in proximity to the endoscope 118 is coupled to the shapesensing optical fiber. The console 650 transmits light through the shapesensing optical fiber and receives light reflected from the shapesensing optical fiber. The shape sensing optical fiber may include asegment of a fiber Bragg grating (FBG). The FBG reflects certainwavelengths of light, while transmitting other wavelengths. The console650 generates reflection spectrum data based on the wavelengths of lightreflected by the FBG.

The data processing module 160 can analyze the reflection spectrum datato generate position and orientation data of the endoscope 118 in two orthree dimensional space. In particular, as the endoscope bends 118, theshape sensing optical fiber embedded inside also bends. The specificwavelengths of light reflected by the FBG changes based on the shape ofthe shape sensing optical fiber (e.g., a “straight” endoscope is in adifferent shape than a “curved” endoscope). Thus, the data processingmodule 160 can determine, for example, how many degrees the endoscope118 has bent in one or more directions (e.g., in response to commandsfrom the surgical robotic system 100) by identifying differences in thereflection spectrum data. Similar to the EM sensor, the shape sensingoptical fiber is suitable for data collection inside the body of thepatient because no line-of-sight to the shape sensing optical fiber isrequired.

V. D. Fluoroscopic Imaging

FIG. 6E is a diagram of a fluoroscopic imaging system 660 in proximityto an endoscope 118 according to one embodiment. The endoscope 118 isinserted by robotic arms 102 into a patient 670 undergoing a surgicalprocedure. The fluoroscopic imaging system 660 is a C-arm that includesa generator, detector, and imaging system (not shown). The generator iscoupled to the bottom end of the C-arm and faces upward toward thepatient 670. The detector is coupled to the top end of the C-arm andfaces downward toward the patient 670. The generator emits X-ray wavestoward the patient 670. The X-ray waves penetrate the patient 670 andare received by the detector. Based on the received X-ray waves, thefluoroscopic imaging system 660 generates the images of body parts orother objects inside the patient 670 such as the endoscope 118. Incontrast to the optical cameras described in Section. V. B. CameraSensors that capture images of the actual endoscope, the fluoroscopicimaging system 660 generates images that include representations ofobjects inside the patient 670, e.g., an outline of the shape of anendoscope based on the reflected X-rays. Thus, the data processingmodule 160 can use similar image processing techniques as previouslydescribed such as optical flow to determine the position and motion ofthe endoscope, e.g., in response to commands from the surgical roboticsystem 100.

VI. Insertion and Roll Offset

FIG. 7A shows a length 700 of a leader 315 of an endoscope 118 extendedoutside of a sheath 311 of the endoscope 118 according to oneembodiment. As the length 700 increases, the flexibility of the distalend of the leader 315 increases because the length 700 is not assignificantly enclosed by the sheath 311. In comparison, the portion ofthe leader 315 radially enclosed by the sheath 311 is less flexiblebecause the material of the sheath 311 provides more rigidity. In someembodiments, since the physical characteristics of the endoscope variesbased on the length of extension, the surgical robotic system 100 needsto provide commands to move the endoscope that account for theextension. For example, the distal end of the endoscope may becomeheavier (and/or more flexible) as the extension increases because thereis more length of the leader outside of the sheath. Thus, to achieve thesame bending movement, the surgical robotic system 100 may need toprovide a command that translates pull wires of the endoscope to agreater degree relative to a command to move an endoscope with a smallerlength of extension.

FIG. 7B shows a relative roll angle 710 of the leader 315 of theendoscope 118 relative to the sheath 311 of the endoscope 118 accordingto one embodiment. The leader 315 and/or the sheath 311 may be moreflexible in certain directions compared to other directions, e.g., dueto variances in the material of the endoscope 118. Thus, based on therelative roll angle 710, the flexibility of the endoscope 118 may changein a certain direction. In addition to accounting for the length ofextension as described above, the surgical robotic system 100 may alsoneed to account for the relative roll angle when providing commands tomove the endoscope. For example, a command to bend the endoscope 90degrees may result in an actual bend of 80 degrees when the relativeroll angle is 5 degrees, but may result in an actual bend of 100 degreeswhen the relative roll angle is −5 degrees.

VII. Calibration

VII. A. Overview

The surgical robotic system 100 performs a calibration process todetermine gain values that compensate for imperfections of anendoscope's behavior. During the calibration process, the surgicalrobotics system 100 moves the endoscope to one or more target positions(or angles) by translating one or more pull wires according to one ormore commands. The surgical robotics system 100 receives spatial dataindicating actual positions and orientations of the endoscope achievedin response to the commands, where the actual positions may be differentthan the target positions due to the endoscope's imperfections. Thesurgical robotics system 100 determines the gain values based on thecommands, the target positions desired to be achieved, and the actualpositions achieved. The surgical robotics system 100 can perform such acalibration process before a surgical procedure, for example, on amanufacturing line for quality assurance, or in a laboratory or clinicalsetting. Additionally, the surgical robotics system 100 can perform sucha calibration process while performing a surgical procedure on apatient.

As a simple illustrative example, the material of a particular endoscopemay be stiffer than expected. When a calibration process is performed,the spatial data indicates that the endoscope deflected to an actualposition of 30 degrees in the pitch direction, whereas the targetposition was 60 degrees. As part of an example calibration process, thesurgical robotics system 100 determines that the corresponding gainvalue for the endoscope is the decimal value of 2.0 because the targetposition is two times the value of the actual position. In otherembodiments, the gain value may be represented using other formats,e.g., a percentage, an integer, in unit of degrees, etc.

Gain values can be associated with a particular pull wire, endoscope,direction of motion (e.g., positive pitch direction, negative yawdirection, or roll direction), and/or other types of factors. Inparticular, the example described above is a trivial scenario thatassumes a constant gain value of 2.0 for all conditions of theendoscope. However, in practice, calibrating endoscopes is a morecomplex problem because the gain values depend on a plethora of factors,either independently or in combination. For instance, the gain value maybe 2.0 in the positive pitch direction, 3.0 in the negative pitchdirection, 1.5 in the positive yaw direction, etc. Further, the gainvalue may be 2.0 in the positive pitch direction for a first pull wirebut 2.2 for a second pull wire in the same endoscope. Additionally, thegain value may be 2.0 for the first pull wire of a first endoscope, but2.5 for the first pull wire of a second endoscope.

In some embodiments, the calibration module 130 receives a length of aleader of the endoscope extended outside of (or radially enclosed by) asheath of the endoscope and/or a relative roll angle of the leaderrelative to the sheath. The calibration module 130 determines the gainvalues further based on the length and/or the relative roll angle. Gainvalues for a certain length or relative roll angle may differ from gainvalues for another length or relative roll angle because the endoscopemay be more flexible in a particular direction or segment of theendoscope.

In one embodiment, the endoscope (e.g., the leader and/or the sheath)includes multiple segments each having a different level of stiffness.The calibration module 130 receives a Young's modulus of at least one ofthe segments and determines the gain values further based on the Young'smodulus.

In one embodiment, a complete calibration process involves severalsub-calibration processes. For example, the surgical robotic system 100provides a command to move an endoscope to a target position in a firstdirection. The calibration module 130 receives calibration dataindicating an actual position of the endoscope, which may differ fromthe target position. The surgical robotic system 100 relaxes theendoscope back to a resting position, and repeats the data collectionprocess for a number of other directions. The surgical robotic system100 can also provide commands to extend the leader to a greater lengthsoutside of the sheath, and repeat the calibration data collectionprocess for a number of different lengths of extension. Similarly, thesurgical robotic system 100 can provide commands to rotate the leader toa relative roll angle relative to the sheath, and repeat the calibrationdata collection process for a number of different relative roll angles.

The calibration module 130 determines gain values based on an aggregatecalibration dataset from each of the sub-calibration processes. Asevident by the number of potential combination of factors to considerduring calibration, the calibration process may become a more complexprocess with many nested loops of different tests. Thus, it isadvantageous to automate the calibration using the surgical roboticsystem 100, e.g., to help keep track of all the factors that need to betested, reduce the chance for calibration errors or oversights, andeliminate the need for a user to manually conduct rote tasks for eachtest.

In some embodiments, the calibration module 130 stores the calibrationdata and associated gain values with one or more other factors (e.g.,information about the corresponding command to move the endoscope, adirection of movement, an identifier of a certain pull wire, a lengthand/or relative roll angle of the leader relative to the sheath, or aunique identifier of the endoscope) in the calibration store 140. Thecalibration module 130 may upload the calibration data, gain values,and/or factors to a global calibration database including informationfrom multiple endoscopes.

VII. B. Calibration Models

The surgical robotic system 100 may use one or more types of models togenerate commands to move an endoscope appropriately based on thecalibration data. In particular, the command module 150 generates acommand for each pull wire of an endoscope based on parameters of one ofthe models, where the parameters and associated gain values aredetermined based on the calibration data. The parameters may be the sameas the gain values for some models, while for other models, the surgicalrobotic system 100 may determine the gain values based on theparameters. The models may be associated with the leader, sheath, orboth the leader and sheath of an endoscope. Embodiments of models thatare associated with both the leader and sheath account for parametersdescribing interaction between the leader and sheath, e.g., the lengthof extension and relative roll angle of the leader relative to thesheath.

In one embodiment, the calibration module 130 uses an empirical modelimplemented with a matrix of gain values. The gain values areempirically determined by solving a set of linear equations based oncalibration data from previously completed calibration processes. Thecalibration module 130 can multiply a vector representing the inputcommand (e.g., including a target translation for each pull wire of anendoscope as well as extension and relative roll values) by the matrixof gain values can generate an output vector representing an adjustedcommand (e.g., including modified translations for one or more of thepull wires). The empirical model gain values may compensate forpull-wire specific compression or expansion based on bending of theendoscope. In particular, the distance that a certain pull wire travelsinside the endoscope may shrink or lengthen based on the curvature ofthe endoscope. In some embodiments, the empirical model accounts for thedependency between wires in opposing directions. For example, a firstwire corresponds to the positive pitch direction and a second wirecorresponds to the negative pitch direction. Providing slack on thefirst wire while pulling on the second wire both contribute to the samemotion of bending the endoscope in the negative pitch direction.

In one embodiment, the calibration module 130 uses a physics based modelto determine the effective physical properties of the endoscope as itbends. The physics based model has the potential to more fully capturethe behavior of the endoscope. As a comparison, a trivial model mayassume that a bent endoscope bends uniformly throughout a particularlength of the endoscope according to a given bending stiffness in thatparticular length and remains straight throughout the rest of the lengthof the endoscope. Further, the physics based model may decompose theleader and sheath of the endoscope into individual segments that eachhave an associated bending stiffness. The physics based model alsoconsiders the stiffness of the pull wires and the effect of the sheathand leader interaction (e.g., extension length and relative roll angle)on the stiffness of any particular segment.

With the physics based model, the computer system 120 may use inversesolid mechanics to translate commands to move the endoscope (e.g.,indicating an angle to bend in pitch and/or yaw directions) intodistances that the surgical robotic system 100 should translate one ormore pull wires to achieve the desired motion. Further, by using thephysics based model, the robotic system 100 may move one or more IDMs tocompensate for any unwanted motion of the endoscope's distal tip as aresult of axial deformations coupled to bending motions.

In one embodiment, the calibration module 130 uses a model-lessinversion process to generate commands to move the endoscope. Forexample, the calibration module 130 implements a lookup table that doesnot require the use of gain values and/or parameters. Instead, thelookup table maps previously recorded input values (e.g., commands tomove the endoscope in pitch and yaw directions) to output values (e.g.,translation for each pull wire of the endoscope) based on calibrationdata. The lookup table may interpolate (e.g., solved via Delaunaytriangulation or other multidimensional triangulations techniques) orextrapolate between data points if the exact input-to-output mapping isnot known, e.g., bending 42 degrees can be interpolated using datapoints for 40 degrees and 45 degrees. To reduce the amount ofcomputational resources required for the computer system 120 to executethe lookup table, the calibration module 130 can minimize the size ofthe data set of mappings by using techniques such as Taylordecomposition or approximating the data set using a Fourierrepresentation.

VII. C. Example Process Flows

FIG. 8A is a flowchart of a process 800 for automated calibration of anendoscope according to one embodiment. The process 800 may includedifferent or additional steps than those described in conjunction withFIG. 8A in some embodiments, or perform steps in different orders thanthe order described in conjunction with FIG. 8A. The process 800 isparticular for calibrating an embodiment of an endoscope including fourpull wires, e.g., each separated by 90 degrees and corresponding to the,positive or negative, pitch or yaw directions. However, the process 800can be generalized to any number of pull wires, and further discussedwith reference to FIG. 8B. Since the computer system 120 is capable ofautomating the process 800, a user does not have to manually perform acalibration procedure to use the surgical robotic system 100. Automatedcalibration is advantageous, e.g., because the process reduces the timerequired to calibrate an endoscope.

The command module 150 provides 804 a command to move an endoscope to atarget position in a first direction. The calibration module 130receives 806 spatial data indicating an actual position and orientationof the endoscope, which moved in response to the command. The spatialdata can be received from spatial sensors (e.g., coupled to theendoscope or positioned in proximity to the endoscope) such asaccelerometers, gyroscopes, fiducial markers, fiber optic cables,cameras, or an imaging system, as previously described in Section V.Spatial Sensors. The spatial data describes the position and/ororientation of the endoscope—or a portion of the endoscope—in one ormore directions of movement. The command module 150 provides 808 acommand to relax the endoscope to a resting position.

The command module 150 provides 810 a command to move the endoscope tothe target position in a second direction. The calibration module 130receives 812 spatial data. The command module 150 provides 814 a commandto relax the endoscope to the resting position.

The command module 150 provides 816 a command to move the endoscope tothe target position in a third direction. The calibration module 130receives 818 spatial data. The command module 150 provides 820 a commandto relax the endoscope to the resting position.

The command module 150 provides 822 a command to move the endoscope tothe target position in a fourth direction. The calibration module 130receives 824 spatial data. The command module 150 provides 826 a commandto relax the endoscope to the resting position.

The target position may remain constant for each of the four directions.In some embodiments, the target position varies between differentdirections. For instance, the target position is 90 degrees for thefirst and third directions and 45 degrees for the second and fourthdirections. The first, second, third, and fourth directions may be thepositive pitch, positive yaw, negative pitch, and negative yawdirections, in any particular order. In other embodiments, the commandsmove the endoscope toward the target position simultaneously in two ormore directions, e.g., 60 degrees in both the positive pitch andpositive yaw directions. Though process 800 involves four directions, inother embodiments, the computer system 120 can repeat the steps 804through 808 for any other number of directions (more or fewer).

The calibration module 130 determines 828 gain values for pull wires ofthe endoscope based on the spatial data for one or more of thedirections. The calibration module 130 may determine a gain valueassociated with each pull wire. At least one of the gain values may havea value different from unity. A unity gain value indicates that thecorresponding pull wire exhibits ideal behavior, e.g., the actual motionof the endoscope matches the target motion based on translation of thecorresponding pull wire. In some embodiments, the calibration module 130retrieves default gain values (e.g., determined in a previouscalibration process) for the pull wires and determines the gain valuesfurther based on the default gain values.

The calibration module 130 stores 830 the gain values in the calibrationstore 140. The endoscope may include a computer readable tangiblemedium, e.g., flash memory or a database, to store the gain values. Insome embodiments, the command module 150 provides a command to modifythe length and/or relative roll angle of the leader relative to thesheath, and the surgical robotic system 100 repeats steps of the process800 to determine gain values associated with the modified length and/orrelative roll angle.

FIG. 8B is a flowchart of a process 840 for automated calibration of anendoscope based on length of extension and relative roll angle accordingto one embodiment. The process 840 may include different or additionalsteps than those described in conjunction with FIG. 8B in someembodiments, or perform steps in different orders than the orderdescribed in conjunction with FIG. 8B. In contrast to the process 800,the process 840 is generalized to any number of directions, as well asany number of lengths of extension and relative roll angles of theleader relative to the sheath of an endoscope. For instance, instead ofan endoscope with four pull wires offset from each other by 90 degrees,an endoscope may include three pull wires offset from each other by 120degrees, or in any other configuration with different offset angles(e.g., at the 11 o'clock, 2 o'clock, and 6 o'clock hand positions of aclock).

The surgical robotic system 100 provides 850 a command to move anendoscope to a length or extension and/or relative roll angle. Thesurgical robotic system 100 performs 860 calibration at the length orextension and/or relative roll angle. In step 860, the command module150 provides 862 a command to move the endoscope to a target position ina direction. The calibration module 130 receives 864 spatial dataindicating an actual position and orientation of the endoscope. Thecommand module 150 provides 866 a command to relax the endoscope to aresting position. The surgical robotic system 100 repeats the steps862-866 for each direction in a set of directions. Further, the surgicalrobotic system 100 repeats the steps 850-860 for each length ofextension and/or relative roll angle (or combination of lengths ofextension and relative roll angles) in a set of different lengths ofextension and/or relative roll angles. The calibration module 130determines 870 gain values based on spatial data received from eachcalibration.

FIG. 9 is a flowchart of a process 900 for intraoperative automatedcalibration of an endoscope to one embodiment. The process 900 mayinclude different or additional steps than those described inconjunction with FIG. 9 in some embodiments, or perform steps indifferent orders than the order described in conjunction with FIG. 9. Insome embodiments, the command console 200 may use the process 900 in thevelocity mode or position control mode previously described in SectionII. Command Console.

The calibration module 130 retrieves 910 default gain values for anendoscope including pull wires, a leader, and a sheath. Each pull wiremay be associated with one of the default gain values. The surgicalrobotic system 100 inserts 920 the endoscope into a patient undergoing asurgical procedure, e.g., ureteroscopy, percutaneous nephrolithotomy(PCNL), colonoscopy, fluoroscopy, prostatectomy, colectomy,cholecystectomy, inguinal hernia, and bronchoscopy. The calibrationmodule 130 receives 930 information about a relative roll angle of theleader relative to the sheath and a length of the leader radiallyenclosed by the sheath. The information about the relative roll angleand the length may be based on previous commands provided to move theendoscope, a default relative roll angle and length value, or datagenerated by sensors (e.g., an accelerometer and gyroscope coupled tothe endoscope). The command module 150 provides 940 a command to movethe endoscope by translating at least one of the pull wires.

The calibration module 130 receives 950 spatial data of the endoscopehaving been moved in response to the command. In one embodiment, thespatial data is received from a fluoroscopic imaging system. Thefluoroscopic imaging system can capture images of the endoscope insidethe patient, which enables the surgical robotic system 100 to performthe process 900 during a surgical procedure. The calibration module 130determines 960 a new gain value based on the spatial data, thecorresponding default gain value, the length of extension, the relativeroll angle, and/or the command. The calibration module 130 stores 970the new gain value in the calibration store 140. The surgical roboticsystem 100 can generate additional commands based on new gain valuesdetermined by the process 900. For example, the endoscope moves to anactual position of 80 degrees in response to a first command, where thetarget position is actually 90 degrees. The command module 150 generatesa new command based on the new gain values and provides the new commandto move the endoscope using the surgical robotic system 100. Since thenew command compensates for the angle discrepancy (that is, 80 degreesis 10 degrees short of 90 degrees), the endoscope moves to an actualposition of 90 degrees in response to the new command.

VIII. Alternative Considerations

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs throughthe disclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various modifications,changes and variations, which will be apparent to those skilled in theart, may be made in the arrangement, operation and details of the methodand apparatus disclosed herein without departing from the spirit andscope defined in the appended claims.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context unlessotherwise explicitly stated.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program product includinga computer-readable non-transitory medium containing computer programcode, which can be executed by a computer processor for performing anyor all of the steps, operations, or processes described.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product mayinclude information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

1-19. (canceled)
 20. A method comprising: retrieving, from computerreadable media storing calibration data, a plurality of default gainvalues each associated with a pull wire of a plurality of pull wires ofan endoscope; inserting the endoscope into a body of a patient using asurgical robotic system; providing a command, derived at least partlyfrom one or more of the plurality of default gain values, to move theendoscope by translating at least one of the plurality of pull wiresusing the surgical robotic system; receiving spatial data indicating anactual position of the endoscope having been moved in response to thecommand; determining, for at least one of the pull wires, a new gainvalue based on the spatial data; and storing the new gain value in thecomputer readable media.
 21. The method of claim 20, further comprising:generating a second command based on the new gain value; and providingthe second command to move the endoscope using the surgical roboticsystem.
 22. The method of claim 20, wherein receiving the spatial datacomprises: positioning a fluoroscopic imaging system in proximity to theendoscope; and capturing a plurality of fluoroscopic images of theendoscope by the fluoroscopic imaging system.
 23. The method of claim20, wherein the surgical robotic system deflects the endoscope to anangle in a yaw direction and a pitch direction in response to thecommand.
 24. The method of claim 20, wherein the endoscope comprises atleast one electromagnetic (EM) sensor coupled to a distal end of theendoscope, and wherein the method further comprises: positioning atleast one EM field generator in proximity to the EM sensor; and whereinreceiving the spatial data comprises detecting, at the EM sensor, an EMfield whose strength is a function of the actual position of the distalend of the endoscope containing the EM sensor.
 25. The method of claim20, wherein the endoscope comprises one or more spatial sensors coupledto a distal end of the endoscope, the one or more spatial sensorsincluding at least one of an accelerometer or a gyroscope, and whereinreceiving the spatial data comprises detecting, by the one or morespatial sensors, motion in at least one direction.
 26. The method ofclaim 20, wherein the endoscope comprises an optical fiber embeddedinside the endoscope, wherein the method further comprises: positioninga console in proximity to the endoscope, the console coupled to theoptical fiber and configured to generate reflection spectrum data basedon light reflected by the optical fiber; and wherein receiving thespatial data comprises analyzing the reflection spectrum data.
 27. Themethod of claim 20, wherein the endoscope includes a camera lens and aworking channel, and wherein the camera lens and the working channel areeach non-concentric to each pull wire of the plurality of pull wires.28. The method of claim 20, wherein the endoscope includes a sheathtubular component and a leader tubular component, the sheath tubularcomponent including a first pull wire of the plurality of pull wires,and the leader tubular component including a second pull wire of theplurality of pull wires.
 29. The method of claim 28, wherein the leadertubular component and the sheath tubular component each include aplurality of segments.
 30. The method of claim 29, wherein the firstpull wire of the plurality of pull wires is spiraled at a first anglealong a first segment of the plurality of segments of the sheath tubularcomponent.
 31. The method of claim 30, wherein the second pull wire ofthe plurality of pull wires is spiraled at a second angle along a secondsegment of the plurality of segments of the leader tubular component.32. The method of claim 28, further comprising: receiving informationindicating a roll angle of the leader tubular component relative to thesheath tubular component and a length of the leader tubular componentradially enclosed by the sheath tubular component; and wherein the newgain value is further determined based on at least one of the roll angleand the length.
 33. The method of claim 20, wherein moving the endoscopeby translating the at least one of the plurality of pull wires using thesurgical robotic system comprises: providing the command to a pluralityof robotic arms coupled to the endoscope.
 34. The method of claim 20,wherein the at least one of the plurality of pull wires includes a firstpull wire and a second pull wire, and wherein providing the commandcomprises: providing a first subcommand to translate the first pull wirea first distance; and providing a second subcommand to translate thesecond pull wire a second distance.
 35. A surgical robotic systemcomprising: one or more robotic arms; an endoscope comprising aplurality of pull wires; non-transitory computer-readable storage mediastoring instructions that when executed by a processor cause theprocessor to perform steps including: retrieve a plurality of defaultgain values each associated with one of the pull wires; insert theendoscope into a body of a patient using the one or more robotic arms;provide a command, derived from one or more of the plurality of defaultgain values, to move the endoscope by translating at least one of theplurality of pull wires using the one or more robotic arms; receivespatial data indicating an actual position of the endoscope having beenmoved in response to the command; determining, for at least one of thepull wires, a new gain value based on the spatial data; and storing thenew gain value.
 36. The surgical robotic system of claim 35, wherein thesteps further include: generating a second command based on the new gainvalue; and providing the second command to move the endoscope using theone or more robotic arms.
 37. The surgical robotic system of claim 35,wherein receiving the spatial data comprises: retrieving a plurality offluoroscopic images of the endoscope captured by a fluoroscopic imagingsystem.
 38. The surgical robotic system of claim 35, wherein the one ormore robotic arms deflects the endoscope to an angle in a yaw directionand a pitch direction in response to the command.
 39. The surgicalrobotic system of claim 35, wherein the endoscope comprises at least oneelectromagnetic (EM) sensor coupled to a distal end of the endoscope,and wherein receiving the spatial data comprises: detecting, at the EMsensor, an EM field whose strength is a function of the actual positionof the distal end of the endoscope containing the EM sensor relative toan EM field generator.
 40. The surgical robotic system of claim 35,wherein the endoscope comprises one or more spatial sensors coupled to adistal end of the endoscope, the one or more spatial sensors includingat least one of an accelerometer or a gyroscope, and wherein receivingthe spatial data comprises detecting, by the one or more spatialsensors, motion in at least one direction.
 41. The surgical roboticsystem of claim 35, wherein the endoscope comprises an optical fiberembedded inside the endoscope, and wherein receiving the spatial datacomprises: analyzing reflection spectrum data generated based on lightreflected by the optical fiber.
 42. The surgical robotic system of claim35, wherein the endoscope includes a camera lens and a working channel,and wherein the camera lens and the working channel are eachnonconcentric to each pull wire of the plurality of pull wires.
 43. Thesurgical robotic system of claim 35, wherein the endoscope comprises asheath tubular component and a leader tubular component, the sheathtubular component including a first pull wire of the plurality of pullwires, and the leader tubular component including a second pull wire ofthe plurality of pull wires.
 44. The surgical robotic system of claim43, wherein the leader tubular component and the sheath tubularcomponent each include a plurality of segments.
 45. The surgical roboticsystem of claim 44, wherein the first pull wire of the plurality of pullwires is spiraled at a first angle along a first segment of theplurality of segments of the sheath tubular component.
 46. The surgicalrobotic system of claim 45, wherein the second pull wire of theplurality of pull wires is spiraled at a second angle along a secondsegment of the plurality of segments of the leader tubular component.47. The surgical robotic system of claim 43, further comprising:receiving information indicating a roll angle of the leader tubularcomponent relative to the sheath tubular component and a length of theleader tubular component radially enclosed by the sheath tubularcomponent; and wherein the new gain value is further determined based onat least one of the roll angle and the length.
 48. The surgical roboticsystem of claim 35, wherein the one or more robotic arms includes atleast a first robotic arm and a second robotic arm, and wherein movingthe endoscope by translating the at least one of the plurality of pullwires using the one or more robotic arms comprises: providing a firstsubcommand to the first robotic arm; and providing a second subcommand,being different than the first subcommand, to the second robotic arm.49. The surgical robotic system of claim 35, wherein the at least one ofthe plurality of pull wires includes a first pull wire and a second pullwire, and wherein providing the command comprises: providing a firstsubcommand to translate the first pull wire a first distance; andproviding a second subcommand to translate the second pull wire a seconddistance.
 50. The method of claim 20, wherein the plurality of defaultgain values are determined during a calibration process.
 51. The methodof claim 50, wherein the plurality of default gain values are based atleast on a discrepancy between a target position and an actual positionof the endoscope.